Skip to content

Refactor legacy field rip-out around canonical compat boundaries#15

Merged
andrewDoing merged 9 commits intomainfrom
rag-cleanup
Mar 19, 2026
Merged

Refactor legacy field rip-out around canonical compat boundaries#15
andrewDoing merged 9 commits intomainfrom
rag-cleanup

Conversation

@andrewDoing
Copy link
Copy Markdown
Owner

@andrewDoing andrewDoing commented Mar 18, 2026

Summary

  • remove legacy top-level field ownership from the core backend and frontend surfaces while keeping compatibility behavior inside the rag_compat boundary
  • standardize canonical question, answer, and reference derivation across backend and frontend helpers
  • centralize frontend rag-compat plugin-envelope writes, refresh generated API artifacts, and update regression coverage for the canonical data shape
  • remove obsolete legacy scripts and related compatibility-only paths that are no longer part of the supported model

Validation

  • make -f Makefile.harness check
  • make -f Makefile.harness api-check
  • make -f Makefile.harness test

Notes

  • Review completed in .copilot-tracking/reviews/2026-03-18/legacy-field-ripout-plan-review.md
  • Follow-up items are optional and limited to a narrow writer unit test plus separate cleanup of pre-existing frontend Biome warnings

andrewDoing and others added 7 commits March 17, 2026 09:21
@andrewDoing
refactor(backend): confine rag compat to references, derive conversat…
22bd3c4 
…ion text from history, route reference access through pack

- remove the core canonical field shim and split generic question and answer access into conversation fields
- normalize rag-compat imports and writes to plugin-owned references while keeping legacy read compatibility
- update backend repos, services, computed tags, demo seed, scripts, and tests to use the new seams

🧭 - Generated by Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
chore: checkpoint legacy field ripout state
35964bd 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
fix: implement legacy ripout review findings
84093cf 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
chore: remove local test artifact
7fe24cd 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
fix(legacy-ripout): address review-2 through review-5 findings
57ad2b4 
Complete the remaining legacy-field ripout remediation across backend, frontend, and tests from multiple review rounds.\n\n- remove remaining generic history.refs coercion paths and tighten core boundary\n- canonicalize demo/reference flows to plugin-owned references\n- harden adapter/model mapping and compat key scrubbing behavior\n- strengthen integration/unit coverage for ownership semantics and canonical counts\n- refresh .copilot-tracking plan/detail/log/changes artifacts with review-5 outcomes\n- re-run focused suites plus check/api-check/test gates

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
refactor: align canonical turn semantics, centralize rag compat writes
f127b9c 
- standardize latest-user question derivation across backend and frontend
- move rag-compat envelope shaping into shared frontend helpers
- tighten canonical compat reference handling and regression coverage

🔒 - Generated by Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
style: apply ruff formatting to backend tests
e7ed7e8 
🔒 - Generated by Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 18, 2026 •
edited
Loading

GTC Backend Unit Test Results

369 tests   - 4   369 ✅  - 4   1s ⏱️ ±0s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 703517b. ± Comparison against base commit af4e78a.

This pull request removes 27 and adds 23 tests. Note that renamed tests count towards both. 
tests.unit.test_history_with_refs ‑ test_history_item_deserialization
tests.unit.test_history_with_refs ‑ test_history_item_with_refs
tests.unit.test_history_with_refs ‑ test_user_history_item_typically_no_refs
tests.unit.test_rag_compat_pack ‑ test_attach_reference_works_on_generic_item
tests.unit.test_rag_compat_pack ‑ test_collect_approval_errors_generic_item_empty
tests.unit.test_rag_compat_pack ‑ test_collect_approval_errors_rag_item_empty
tests.unit.test_rag_compat_pack ‑ test_detach_reference_nonexistent_url_is_noop
tests.unit.test_rag_compat_pack ‑ test_export_transform_projects_retrieval_candidates_to_refs
tests.unit.test_rag_compat_pack ‑ test_rag_compat_data_contains_synth_question
tests.unit.test_rag_compat_pack ‑ test_rag_compat_data_populated_for_rag_item
…

tests.unit.test_conversation_fields ‑ test_question_and_answer_derivation_follow_user_vs_non_user_contract
tests.unit.test_conversation_fields ‑ test_role_helpers_use_strict_user_semantics
tests.unit.test_cosmos_repo ‑ test_emulator_unicode_normalization_does_not_base64_encode_legacy_refs
tests.unit.test_cosmos_repo ‑ test_emulator_unicode_normalization_encodes_canonical_reference_content
tests.unit.test_history_with_refs ‑ test_history_item_deserialization_rejects_refs
tests.unit.test_history_with_refs ‑ test_history_item_rejects_refs
tests.unit.test_history_with_refs ‑ test_user_history_item_rejects_refs
tests.unit.test_rag_compat_pack ‑ test_attach_reference_writes_owned_references_key
tests.unit.test_rag_compat_pack ‑ test_collect_approval_errors_are_empty
tests.unit.test_rag_compat_pack ‑ test_export_transform_projects_references_and_count
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 18, 2026 •
edited
Loading

GTC Frontend Test Results

301 tests  +10   301 ✅ +10   5s ⏱️ ±0s
 44 suites ± 0     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 703517b. ± Comparison against base commit af4e78a.

This pull request removes 27 and adds 37 tests. Note that renamed tests count towards both. 
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > legacy single-turn conversion > assigns legacy top-level refs to messageIndex 1
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > legacy single-turn conversion > creates 2-turn history from editedQuestion and answer
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > legacy single-turn conversion > creates empty agent turn when answer is empty
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > legacy single-turn conversion > falls back to synthQuestion when editedQuestion is empty
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > multi-turn item top-level refs > assigns top-level refs to undefined messageIndex for true multi-turn
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > reference mapping > assigns turn refs to correct messageIndex
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > reference mapping > maps ref fields correctly
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > role mapping > maps history role 'assistant' to 'agent'
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthToPatch > basic field mapping > includes answer and editedQuestion
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthToPatch > reference handling > includes refs only on agent turns in history
…

tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > multi-turn item top-level refs > does not import compat refs for true multi-turn
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > multi-turn item top-level refs > treats explicit empty canonical references as authoritative
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > reference mapping > ignores retired turn-level refs from history payloads
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > reference mapping > maps canonical plugin reference fields correctly
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > reference mapping > reads canonical rag-compat data.references
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > retired single-turn compat behavior > does not create synthetic turns when answer is empty
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > retired single-turn compat behavior > does not fall back to synthQuestion when editedQuestion is empty
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > retired single-turn compat behavior > does not import legacy top-level refs
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > retired single-turn compat behavior > does not synthesize history from editedQuestion and answer
tests/unit/adapters/apiMapper.test.ts ‑ groundTruthFromApi > retired single-turn compat behavior > treats explicit API empty history as authoritative over compat question/answer
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 18, 2026 •
edited
Loading

GTC Backend Integration Test Results

113 tests   - 27   112 ✅  - 27   43s ⏱️ -6s
  1 suites ± 0     1 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 703517b. ± Comparison against base commit af4e78a.

This pull request removes 28 and adds 1 tests. Note that renamed tests count towards both. 
tests.integration.test_etag_and_refs_cosmos ‑ test_curator_put_refs_with_etag
tests.integration.test_ground_truths_reference_count ‑ test_ground_truth_item_includes_total_references_field
tests.integration.test_ground_truths_reference_count ‑ test_total_references_counts_both_levels
tests.integration.test_ground_truths_reference_count ‑ test_total_references_counts_history_level_refs_only
tests.integration.test_ground_truths_reference_count ‑ test_total_references_counts_item_level_refs_only
tests.integration.test_ground_truths_reference_count ‑ test_total_references_multiple_items_independent
tests.integration.test_ground_truths_reference_count ‑ test_total_references_zero_when_no_refs
tests.integration.test_ground_truths_reference_search ‑ test_ref_url_search_case_sensitive
tests.integration.test_ground_truths_reference_search ‑ test_ref_url_search_combined_with_other_filters
tests.integration.test_ground_truths_reference_search ‑ test_ref_url_search_empty_string_ignored
…

tests.integration.test_etag_and_refs_cosmos ‑ test_curator_put_plugins_with_etag

♻️ This comment has been updated with latest results.

andrewDoing and others added 2 commits March 18, 2026 14:58
@andrewDoing
test(backend): align integration tests with canonical schema
9e03207 
Migrate backend integration fixtures and assertions from legacy synthQuestion/answer/refs payloads to canonical history-based payloads.

Remove rag-compat-coupled integration suites and rework remaining plugin coverage to use generic plugin payloads.

Fix assignment duplication by excluding computed fields during model re-validation.

Validation: make -f Makefile.harness backend-integration-test (112 passed, 1 skipped).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing
style(backend): apply integration test formatting
703517b 
Apply formatter updates produced by repository hooks for backend integration test files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@andrewDoing andrewDoing merged commit 5ca9451 into main Mar 19, 2026
6 checks passed
@andrewDoing andrewDoing deleted the rag-cleanup branch March 19, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrewDoing